Text suggestions for images

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for receiving image data corresponding to an image, processing the image data to identify one or more features within the image, generating one or more keywords based on each of the one or more features, transmitting the one or more keywords to a computing device for displaying a list of the one or more keywords to a user, receiving text, the text comprising at least one keyword of the one or more keywords, that at least one keyword having been selected by the user from the list, and transmitting the image and the text for display, the text being associated with the image.

TECHNICAL FIELD

This specification generally relates to suggesting text for images.

BACKGROUND

People take photographs (photos) to document events and to keepmemories. People often share the photos with friends and family. Inrecent years, digital photography has become more mainstream. Usingdigital photography, a photographer can capture a photograph and storethe photograph as a digital file. The digital file can be stored tocomputer-readable memory, can be copied and can be electronicallydistributed. The Internet has made the sharing of photos much easier.People can email photos to friends, or post them on websites for othersto view. Social networking websites are also used to share photos withfriends and acquaintances. People can also label photos with captions orkeywords.

SUMMARY

In general, innovative aspects of the subject matter described in thisdisclosure may be embodied in methods that include the actions ofreceiving image data corresponding to an image, processing the imagedata to identify one or more features within the image, generating oneor more keywords based on each of the one or more features, transmittingthe one or more keywords to a computing device for displaying a list ofthe one or more keywords to a user, receiving text, the text includingat least one keyword of the one or more keywords, that at least onekeyword having been selected by the user from the list, and transmittingthe image and the text for display, the text being associated with theimage.

These and other implementations may each optionally include one or moreof the following features. For instance, the image data further includesmetadata; the metadata includes geo-location data corresponding to ageographic location where the image was generated; the metadata includestime data corresponding to a time when the image was generated;generating the one or more keywords is further based on the metadata;the one or more features include a landmark; the one or more featuresinclude people; the people include users of a social networking service;the one or more features are objects; a certainty score is generated foreach of the one or more features; the one or more features are rankedbased on the certainty score of each of the one or more features; athreshold number of the one or more features is selected based on theranking; and one or more features with a certainty score that meets athreshold certainty score are included in the ranking.

Other implementations of this aspect include corresponding systems,apparatus, and computer programs, configured to perform the actions ofthe methods, encoded on computer storage devices.

Implementations of the present disclosure provide one or more of thefollowing example advantages: enhancing the use of computing devices,such as mobile computing devices, by reducing an amount of typing theuser performs to provide textual posts, providing richer social poststhrough the addition of meaningful content to posts, suggested keywordsused within a post can provide improved context for other systems (e.g.,post matching, searching), and/or feedback for a recognition engine(e.g., if the user chooses suggested keywords for a post and addsfurther own words, the added words can provide clues for the recognitionengine to rank other keywords with higher certainty in the future).

The details of one or more implementations of the subject matterdescribed in this specification are set forth in the accompanyingdrawings and the description below. Other potential features, aspects,and advantages of the subject matter will become apparent from thedescription, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an example system that can execute implementations of thepresent disclosure.

FIGS. 2A-2C depicts example screenshots in accordance withimplementations of the present disclosure.

FIG. 3 is a flowchart illustrating an example process for generatingsocial post suggestions based on information extracted from an image.

FIG. 4 is a flowchart illustrating an example process for generatingsocial post suggestions based on information extracted from an image.

Like reference numbers represent corresponding parts throughout.

DETAILED DESCRIPTION

Implementations of the present disclosure are generally directed tosuggesting keywords for captions or tags for images. A user can generatedigital images (e.g., using a digital camera) that can each beelectronically stored in an image file. The image files can be uploadedto one or more servers for publication of the images to a viewingaudience. For example, the user can upload the image files to a socialnetworking website for publication to other users of the socialnetworking website. As another example, the user can upload the imagefiles to a photo-sharing website for publication to other users of thephoto-sharing website. Data corresponding to an image can be processedto determine features that may be seen within the image. Examplefeatures can include landmarks identified in the image, objectsidentified in the image, and people recognized in the image. The systemcan also use metadata of the underlying image file, such as timestampdata of when the image was generated and geo-location data indicatingwhere the image was generated. Geo-location can include globalpositioning system (GPS) data, Wifi location data and/or cell towerlocation data.

In some implementations, the keywords can be generated based on thefeatures determined from each image and/or the metadata corresponding tothe image. For example, keywords can include the landmarks, objects, orpeople recognized in the image. The keywords can also include times ofday and/or locations based on the metadata or other data from the imagefile. The keywords can be suggested to the user to be used as keywords,or tags, to be associated with the image when it is published. In someimplementations, the keywords or the features determined can be assigneda score, such as a certainty score, corresponding to a confidencemeasure of the extracted feature. A threshold number of keywords or anumber of keywords with a threshold score can be suggested to the user.For example, the top five likely keywords can be provided to the user assuggested keywords.

By way of non-limiting example, a user can take a picture of the Statueof Liberty at noon using a digital camera (e.g., a stand-alone digitalcamera, a digital camera integrated into a mobile computing device). Theimage data underlying the image can be analyzed to determine the Statueof Liberty as a recognized landmark. A timestamp of the image can beprovided in the image data and can indicate that the photo was taken atnoon. The image data can also include geo-location data indicating thatthe photo was taken in New York City and/or a particular location withinNew York City (e.g., Liberty Island). Alternatively or additionally, theStatue of Liberty can be used to determine that the image was generatedin New York City. Example keywords generated from the features caninclude “Statue of Liberty,” “lunch time,” “New York” and “New YorkCity”. Keywords can include phrases, such as the three example keywords.The keyword “lunch time” can be generated based on the timestamp dataindicating that the image was generated at noon. In someimplementations, additional information (e.g., facts) can be suggestedto the user, which the user can include in a post. Continuing with theexample above, example facts can include historical and/or interestingfacts about the Statue of Liberty (e.g., year built, anniversary dates,height, amount of material used in construction, etc.).

The user can select from the suggested keywords to tag the image orprovide a caption for the image for publication, for example, on thesocial networking site. For example, the user can take and upload animage from a mobile computing device, such as a smart phone. Byreceiving suggested keywords, the user does not have to type outkeywords to tag the image. The selected keywords can be included in atextbox, so that the user can add additional text or edit the suggestedkeywords to tag the image and/or add a caption to the image. In someimplementations, suggested facts can be selected by the user to add tothe post.

In some implementations, for situations in which the systems discussedherein collect personal information about users, the users may beprovided with an opportunity to opt in/out of programs or features thatmay collect personal information (e.g., information about a user'spreferences, information relating to locations the user may have been totake photos, information relating to people recognized in photos, or auser's contributions to social content providers). In addition, in someimplementations, certain data may be anonymized in one or more waysbefore it is stored or used, so that personally identifiable informationis removed. For example, a user's identity may be anonymized so thatidentified user preferences or user interactions are generalized (forexample, generalized based on user demographics) rather than associatedwith a particular user.

FIG. 1 depicts an example system 100 that can execute implementations ofthe present disclosure. The system 100 includes a computing device 102that can communicate with one or more server systems 104 over a network106. The computing device 102 includes an associated user 108. Thenetwork 106 can include a large computer network, such as a local areanetwork (LAN), wide area network (WAN), the Internet, a cellularnetwork, or a combination thereof connecting any number of mobilecomputing devices, fixed computing devices, and server systems. Theserver system 104 includes a computing device 110 and a machine-readablerepository, or database 112. It is appreciated that, although a serversystem 104 is depicted, the server system can represent one or moreserver systems.

In the example system 100, the computing device 102 is illustrated as amobile computing device. It is appreciated, however, that the computingdevice 102 can include any appropriate type of computing device such asa tablet computing device, a desktop computer, a laptop computer, ahandheld computer, a personal digital assistant (PDA), a cellulartelephone, a network appliance, a camera, a smart phone, an enhancedgeneral packet radio service (EGPRS) mobile phone, a media player, anavigation device, an email device, a game console, or a combination ofany two or more of these data processing devices or other dataprocessing devices.

The computing device 102 enables the user 108 to take a photograph(photo) to generate a digital image or access and view an existingimage. The image can be stored as an image file on the computing device102 or accessed through the network 106 from a different computingdevice. The computing device 102 enables the user 108 to add descriptivewords, or tags, to apply to the image. In some implementations, thecomputing device 102 can communicate over the network 106 to provide theimage file to the server system 104. The server system 104 can analyzeimage data of the image file to extract features that may be viewable inthe image. The features can be used to provide suggested keywords thatcan be used as tags and/or a caption for the image. Otherimplementations are possible. In alternative implementations, thephotograph can be generated using a separate device (e.g., a standalonedigital camera, a digital scanner) and uploaded to the server system 104using the computing device 102, for example.

FIGS. 2A-2C depict example screenshots in accordance withimplementations of the present disclosure. For purposes of illustration,the following discussion includes reference to the Statue of Liberty inNew York City as an example feature within a digital image. It isappreciated that implementations of the present disclosure areapplicable to any features that may be depicted in a digital image.

Referring to FIG. 2A, a user can use a mobile computing device 202 tocapture an image 204 of the Statue of Liberty. For example, the mobilecomputing device 202 can include a camera, with which the user cancapture the image 204. Alternatively or additionally, the image 204 canbe accessed by or transferred to the mobile computing device 202 fromdigital memory within the mobile computing device 202 and/or fromanother computing device. For example, the mobile computing device 202can access the image 204 from a social networking site, the image 204can be sent to the user from another user, or the image 204 can betransferred to the mobile computing device 202 from an image collectionstored on a separate computing device. While a mobile computing deviceis depicted in FIGS. 2A-2C, implementations of the present disclosurecan be achieved on any appropriate computing device, including laptopsand desktop computers.

The user can bring up a menu of options provided as iconic buttons 206,208, 210 as shown in FIG. 2B. For example, the menu can be accessed bytapping on the screen of the mobile computing device 202, on the image204 or on a physical button of the mobile computing device 202. Thebuttons 206, 208, 210 can provide the user options to apply to the image204. For example, an “Email Image” button 206 can enable the user toemail the image 204. A “Use as Wallpaper” button 208 can set the image204 as a wallpaper or background image of the mobile computing device202. A “Post to Social Network” button 210 can enable the user to postthe image 204 to a social networking site. The menu can include morebuttons for additional options to apply to the image 204, or lessbuttons. In some implementations, the user can be working directly in asocial networking application executed on the mobile computing device,can take a picture using functionality provided in the social networkingapplication, and can automatically be prompted to share the picturewithout going through the example menu options discussed above.

If the user selects the “Post to Social Network” button 210, the mobilecomputing device 202 can upload the image 204 to a social networkingsite by transmitting the image file to a server system operated by thesocial networking site. The server system can receive the image file andprocess the image data to extract features that may be viewable in theimage 204. For example, the image data of the image 204 can be analyzedto determine that the Statue of Liberty is viewable within the image204. The image data can also include metadata, such as a timestamp. Forexample, the image 204 could have been generated at 12.03 PM as shown onthe mobile computing device 202. The metadata can also includegeo-location data, for example, indicating a location or approximatelocation of where the image 204 was generated (e.g., New York City,Liberty Island). Other features can include objects, such as buildings,cars, or animals. Features can also include people. In someimplementations, the image data can be analyzed to determine that aperson is present in the image. If it is determined that a person ispresent in the image, the image can be further analyzed to determine anidentity of the person. For example, an image can include a friend ofthe user, the friend being socially connected to the user through asocial networking service. In some implementations, the friend can grantpermission to be identified in images submitted by people that thefriend is connected to in the social networking service.

Any appropriate feature extraction engine can be used to extractfeatures from the image. For example, a feature extraction enginedirected to recognizing famous works of art can be used to extractfamous works of art that are viewable in images. As another example, afeature extraction engine directed to recognizing landmarks can be usedto extract landmarks that are viewable in images. As another example, afeature extraction engine directed to recognizing the presence of aperson that is viewable within an image and the identity of the personcan be used. In some implementations, images can be analyzed usingmultiple feature extraction engines. In some implementations, images canbe analyzed using a single feature extraction engine. Each featureextraction engine can be provided as a computer program product that isexecutable on a computing device. In some implementations, the featureextraction engine can be executed using one or more server systems(e.g., the server system 104 of FIG. 1), and/or on a computing device(e.g., the computing device 102 of FIG. 1).

Continuing with the example above, the server system can generatekeywords related to the features extracted from the image. For example,keywords can include “Statue of Liberty,” “Lady Liberty,” and “NewColossus,” generated from the recognized landmark in the photo 204. Thekeywords can also include “New York, N.Y.,” “NYC,” and “New York City.”The location keywords can be generated from geo-location metadata,and/or from recognizing that the Statue of Liberty is located in NewYork City on Liberty Island. The keywords can also include “Lunch” and“Mid-day,” generated from the timestamp metadata. These and otherkeywords can be generated using the features extracted from the photo204.

In some examples, keywords can include one or more keywords that arepre-determined as corresponding to an extracted feature. For example,the extracted feature can be identified as the Statue of Liberty. Anindex of keywords associated with the Statue of Liberty can be accessedto provide one or more keywords (e.g., Lady Liberty, New Colossus,Liberty Island).

In some implementations, keywords can be generated based on a searchquery that includes the extracted feature(s). In some examples, a searchquery can be generated and can include one or more search terms thatcorrespond to one or more extracted features. The search query can beprocessed using conventional searching techniques to generate one ormore search results. The search results can be processed to extract oneor more keywords. Continuing with the example, the extracted feature caninclude “Statue of Liberty.” “Statue of Liberty” can be provided as asearch query and can be input to a search engine. One or more searchresults can be generated and can include information (e.g., contentprovided in one or more web pages) corresponding to the Statue ofLiberty. The information can be processed to extract one or morekeywords.

The keywords or a subset of the keywords can be provided to the mobilecomputing device 202. The mobile computing device 202 can display thekeywords suggested by the server system. In FIG. 2C, the mobilecomputing device 202 receives the suggested keywords from the serversystem and displays the suggested keywords in a box 222. The mobilecomputing device 202 can provide a text box 220 where the user can typein a caption for the image 204 to post with the image 204 when the image204 is published. The user can select among the suggested keywords224-238 to include in the caption. For example, the user can type theword “At” into the text box 220 (e.g., using a touch-screen keyboard notshown in FIG. 2C), and can select the “Lady Liberty” keyword 226. Theuser can continue to type in the text box 220 the word “in,” followed byselecting the “NYC” keyword 234, type the word “for” in the text box220, and select the “Lunch” keyword 236. Thus, the caption would read“At Lady Liberty in NYC for Lunch.”

By providing the suggested keywords, the user can generate a caption forthe picture with fewer clicks on the mobile computing device 202. Forexample, typing the full example caption (At Lady Liberty in NYC forLunch) could require 32 clicks, for example, for all the characters andspaces. By clicking on the suggested keywords instead of typing out theselected keywords, the caption can be generated using 10 clicks insteadof 32. Consequently, adding captions to images is made easier andquicker through implementations of the present disclosure, and users maybe more inclined to add captions and tags to shared images. Further, thesuggested keywords may include keywords that the user may not have usedor thought of, and or that are associated with aspects of the image thatthe user may not have noticed or remembered. As discussed above,suggested facts can be presented to the user, which facts can beselected for inclusion as part of the post.

In some implementations, the image can include a plurality of featuresthat may be extracted from the image. Each feature can include a scoreassociated therewith. In some implementations, the score can include acertainty score, corresponding to how confident the system (e.g., thefeature extraction engine) is that the feature has been correctlyidentified in the image. For example, and continuing with the example ofFIGS. 2A-2C, the system may be fairly certain that the landmark in theimage 204 is the Statue of Liberty. Consequently, the “Statue ofLiberty” feature and keywords generated based on the feature can begiven a high certainty score, such as 0.9 on a scale of 0 to 1, forexample, if the system is 90% confident that the landmark in the image204 is the Statue of Liberty. Other features may also be extracted fromthe image, with each feature including an associated certainty score. Insome implementations, the features can be ranked in order of certaintyscore and a subset of features can be selected for keyword suggestion.For example, the features with the highest X certainty scores (e.g.,X=3, 4 or 5) can be selected, and keywords can be generated based on theselected features.

As a non-limiting example, an image can be analyzed and the followingfeatures with the associated certainty scores can be extracted:

Feature Certainty Score A 0.50 B 0.93 C 0.85 D 0.61 E 0.63Using this example, the features can be ordered based on certainty scorefrom highest to lowest to provide the following ranked order:

-   -   Feature B    -   Feature C    -   Feature E    -   Feature D    -   Feature A        The top three, for example, features can be selected for keyword        suggestion. Continuing with the above example, Feature B,        Feature C and Feature E would be selected and can be processed        to generate suggested keywords.

In some implementations, a threshold certainty score can be used tofurther limit the features that are selected for keyword suggestion. Inthis manner, even though a particular feature may be in the list offeatures having the highest X certainty scores (e.g., X=3, 4 or 5), theparticular feature may still not be selected for keyword suggestion ifthe associated certainty score is less than the threshold certaintyscore. Continuing with the example above, and using a particularthreshold certainty score, Feature E, Feature D and Feature A mayinclude certainty scores that are less than the particular thresholdcertainty score. Consequently, Feature B and Feature C would be the onlyfeatures that are selected for keyword suggestion.

In some implementations, and for each selected feature, one or morekeywords can be identified. The number of keywords provided for eachfeature can be limited to a threshold number of keywords. In someimplementations, each identified keyword can include an associatedpopularity score that can reflect how common and/or recognizable akeyword may be. In some implementations, the keywords can be ranked inorder of popularity score and a subset of keywords can be selected assuggested keywords that are to be presented to the user. For example,the keywords with the highest X popularity scores (e.g., X=3, 4 or 5)can be selected as suggested keywords. In some implementations, othermetrics can be used to rank the keywords. In some examples, a landmarkrecognized within an image can be weighted higher than an object (e.g.,a car) and/or time of day. In this manner, more suggested keywordscorresponding to an identified landmark may be provided than areprovided for the object and/or the time of day, for example.

In some implementations, no keywords can be suggested if features cannotbe extracted or keywords cannot be generated with sufficient certainty.In such instances, a simple message (e.g., “No Keyword Suggestions”) canbe displayed to the user.

FIG. 3 is a flowchart of an example process 300 for generating socialpost suggestions based on information extracted from an image. In someimplementations, actions represented in the example process 300 may beperformed by a computing system (e.g., the computing device 102 of FIG.1).

A digital image is captured (302). The computing device 102 of FIG. 1can be used to take a photo, or a digital image, for example. Thedigital image can include features such as landmarks, objects, and/orpeople that are viewable within the image. An image file including imagedata is created (304). The captured digital image can be stored on thecomputing device 102 as an image file including image data, such thatthe image can be presented on a screen of the computing device 102. Asone option, metadata can be appended to the image file (306). Themetadata can include timestamp and geo-location data respectivelycorresponding to when and where the image was captured.

It can be determined whether to post the image (308). For example, userinput can be received and can indicate a command from the user to postthe image (e.g., a user click on the iconic button 210 of FIG. 2B). Theimage can be posted, for example, on a social networking site. The usercan decide not to post the image, in which case the process ends. Theuser can also choose to post the image, in which case the image file istransmitted (310). The image file can be transmitted to a server systemfor extraction of features and/or generation of keywords.

Keywords are received (312). The keywords can include descriptions ofthe features of the image, such as landmarks, objects, people, times ofday, and locations. The keywords are displayed (314) to the user. Theuser can use the keywords to generate caption text (316). The captiontext can include selected keywords and other text input by the user. Thecaption text is transmitted (318). The caption text can be associatedwith the image and published on a social networking site for the userand other people, such as the user's connections, to see.

It is appreciated that the process 300 of FIG. 3 is an example processand implementations of the present disclosure can be realized usingother processes. In some examples, an image can be captured and theimage can be uploaded to a backend server system as a backgroundfunction. The user can type in post text and submit the post. In someexamples, the image is already uploaded to the backend server and thepost goes through almost instantly. In some examples, the user maycancel posting the image and, in response, the image is deleted from thebackend server system and the upload is canceled. In some examples,because the image is uploaded in the background, the list of suggestedkeywords can be received back from the backend server system, while useris composing the post and before the user submits the posts. In someexamples, if the suggested keywords have not been provided to the userbefore the user submits the post, the post goes through as normal (i.e.,the user is not blocked or hindered in posting).

FIG. 4 is a flowchart of an example process 400 for generating socialpost suggestions based on information extracted from an image. In someimplementations, actions represented in the example process 400 may beperformed by one or more computing system (e.g., the computing device102 and/or the server system 104 of FIG. 1).

An image file is received (402). The image file can be received by theserver system 104 of FIG. 1 from the computing device 102, for example.Features are extracted (404) from the image file. Example features caninclude objects, landmarks, logos, faces in the image, as well as timeof day and location the image was captured.

The computing system 110 determines whether multiple features areextracted (406). If only one feature is extracted, the process continuesto keyword generation. In some implementations, a certainty score of theextracted feature can be processed to ensure a threshold level ofcertainty of the extracted feature. If multiple features are extracted,a certainty score is generated for each feature (408). The certaintyscore can correspond to a confidence that the feature extracted from theimage is, in fact, in the image.

The features are ordered based on certainty scores (410). In someimplementations, only features with a minimum threshold certainty scoreare included in the ordering. The top X features are selected (412). Xcan be any appropriate number of features. For example, the user can sethow many features are selected.

Keywords are generated based on the features (414). Each feature can bethe basis for one or more keywords. The number of keywords generated foreach feature can vary, for example, depending on how many features wereextracted from the image. As discussed herein, each keyword can includean associated score, which can also vary.

The keywords are transmitted (416) to the user. The user can selectamong the keywords to generate a caption for the image. The user cantransmit the caption text, and the caption text can be received (418).The image is published with the caption text (420), for example, on aprofile associated with the user on the social networking site.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. For example, various formsof the flows shown above may be used, with steps re-ordered, added, orremoved. Accordingly, other implementations are within the scope of thefollowing claims.

Implementations of the present disclosure and all of the functionaloperations provided herein can be realized in digital electroniccircuitry, or in computer software, firmware, or hardware, including thestructures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them. Implementationsof the invention can be realized as one or more computer programproducts, i.e., one or more modules of computer program instructionsencoded on a computer readable medium for execution by, or to controlthe operation of, data processing apparatus. The computer readablemedium can be a machine-readable storage device, a machine-readablestorage substrate, a memory device, a composition of matter affecting amachine-readable propagated signal, or a combination of one or more ofthem. The term “data processing apparatus” encompasses all apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, or multiple processors or computers.The apparatus can include, in addition to hardware, code that creates anexecution environment for the computer program in question, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a stand alone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this disclose can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. However, a computerneed not have such devices. Moreover, a computer can be embedded inanother device, e.g., a mobile telephone, a personal digital assistant(PDA), a mobile audio player, a Global Positioning System (GPS)receiver, to name just a few. Computer readable media suitable forstoring computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices; magnetic disks, e.g., internal hard disks or removable disks;magneto optical disks; and CD ROM and DVD-ROM disks. The processor andthe memory can be supplemented by, or incorporated in, special purposelogic circuitry.

To provide for interaction with a user, implementations of the inventioncan be implemented on a computer having a display device, e.g., a CRT(cathode ray tube) or LCD (liquid crystal display) monitor, fordisplaying information to the user and a keyboard and a pointing device,e.g., a mouse or a trackball, by which the user can provide input to thecomputer. Other kinds of devices can be used to provide for interactionwith a user as well; for example, feedback provided to the user can beany form of sensory feedback, e.g., visual feedback, auditory feedback,or tactile feedback; and input from the user can be received in anyform, including acoustic, speech, or tactile input.

Implementations of the present disclosure can be realized in a computingsystem that includes a back end component, e.g., as a data server, orthat includes a middleware component, e.g., an application server, orthat includes a front end component, e.g., a client computer having agraphical user interface or a Web browser through which a user caninteract with an implementation of the present disclosure, or anycombination of one or more such back end, middleware, or front endcomponents. The components of the system can be interconnected by anyform or medium of digital data communication, e.g., a communicationnetwork. Examples of communication networks include a local area network(“LAN”) and a wide area network (“WAN”), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this disclosure contains many specifics, these should not beconstrued as limitations on the scope of the disclosure or of what maybe claimed, but rather as descriptions of features specific toparticular implementations of the disclosure. Certain features that aredescribed in this disclosure in the context of separate implementationscan also be provided in combination in a single implementation.Conversely, various features that are described in the context of asingle implementation can also be provided in multiple implementationsseparately or in any suitable subcombination. Moreover, althoughfeatures may be described above as acting in certain combinations andeven initially claimed as such, one or more features from a claimedcombination can in some cases be excised from the combination, and theclaimed combination may be directed to a subcombination or variation ofa subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

Thus, particular implementations of the present disclosure have beendescribed. Other implementations are within the scope of the followingclaims. For example, the actions recited in the claims can be performedin a different order and still achieve desirable results.

What is claimed is: 1-20. (canceled)
 21. A system comprising: acomputing device; and a computer-readable medium coupled to thecomputing device and having instructions stored thereon which, whenexecuted by the computing device, cause the computing device to performoperations comprising: receiving image data corresponding to an image;determining at least one select feature of the image by identifying twoor more different features within the image, generating a certaintyscore for each of the different feature, ranking the different featuresbased on the respective certainty score, and selecting the at least oneselect feature from the different features based on the ranking;generating one or more keywords based on each of the at least one selectfeature; transmitting the one or more keywords to a remote computingdevice for displaying a suggestion list of the one or more keywords to auser; receiving text from the user, the text comprising at least one ormore user-input characters and at least one keyword of the one or morekeywords selected by the user from the suggestion list; and associatingthe text with the image.
 22. The system of claim 21, wherein the text isa caption and wherein associating of text with the image is bypublishing the caption and the image on a social networking site. 23.The system of claim 21, wherein the ranking includes different featuresmeeting a threshold certainty score.
 24. The system of claim 21, whereinat least one of the keywords is generated from geo-location data for theimage and describes a location where the image was generated.
 25. Thesystem of claim 21, wherein at least one of the keywords is generatedfrom time data for the image and describes a time when the image wascreated.
 26. The system of claim 21, wherein the keywords are orderedand wherein a top threshold number of keywords is transmitted.
 27. Thesystem of claim 26, wherein the ordering of keywords is based on aweighting of one or more type of features.
 28. A nontransitory computerstorage medium encoded with a computer program, the program comprisinginstructions that when executed by one or more computers cause the oneor more computers to perform operations comprising: receiving image datacorresponding to an image; determining at least one select feature ofthe image by identifying two or more different features within theimage, generating a certainty score for each of the different feature,ranking the different features based on the respective certainty score,and selecting the at least one select feature from the differentfeatures based on the ranking; generating one or more keywords based oneach of the extracted features; transmitting the one or more keywords toa remote computing device for displaying a list of the one or morekeywords to a user; receiving text from the user, the text comprising atleast one or more user-input characters and at least one keyword of theone or more keywords selected by the user from the list; and associatingthe text with the image.
 29. The computer storage medium of claim 28,wherein the text is a caption and wherein associating of text with theimage is by publishing the caption and the image on a social networkingsite.
 30. The computer storage medium of claim 28, wherein the rankingincludes different features meeting a threshold certainty score.
 31. Thecomputer storage medium of claim 28, wherein at least one of thekeywords is generated from geo-location data for the image and describesa location where the image was generated.
 32. The computer storagemedium of claim 28, wherein at least one of the keywords is generatedfrom time data for the image and describes a time when the image wascreated.
 33. The computer storage medium of claim 28, wherein thekeywords are ordered and wherein a top threshold number of keywords istransmitted.
 34. The computer storage medium of claim 33, wherein theordering of keywords is based on a weighting of one or more type offeatures.
 35. A computer-implemented method comprising: receiving imagedata corresponding to an image; receiving image data corresponding to animage; determining at least one select feature of the image byidentifying two or more different features within the image, generatinga certainty score for each of the different feature, ranking thedifferent features based on the respective certainty score, andselecting the at least one select feature from the different featuresbased on the ranking; generating one or more keywords based on each ofthe extracted features; transmitting the one or more keywords to aremote computing device for displaying a list of the one or morekeywords to a user; receiving text from the user, the text comprising atleast one or more user-input characters and at least one keyword of theone or more keywords selected by the user from the list; and associatingthe text with the image.
 36. The method of claim 35, wherein the text isa caption and wherein associating of text with the image is bypublishing the caption and the image on a social networking site. 37.The method of claim 35, wherein the ranking includes different featuresmeeting a threshold certainty score.
 38. The method of claim 35, whereinat least one of the keywords is generated from geo-location data for theimage and describes a location where the image was generated.
 39. Themethod of claim 35, wherein at least one of the keywords is generatedfrom time data for the image and describes a time when the image wascreated.
 40. The method of claim 35, wherein the keywords are orderedand wherein a top threshold number of keywords is transmitted.